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This  report  details  the  pilot  administration  of  a  Job  Performance  Measurement 
Package  designed  to  assess  first-term  U.S.  Marine  Corps  (USMC)  3-79  jet  engine  mechanic 
job  proficiency.  The  data  collection  reported  herein  completes  an  important  inter-service 
transfer  of  testing  technology,  a  cooperative  effort  between  the  Air  Force  and  the  Navy. 

Information  contained  herein  is  intended  to  benefit  the  Navy  and  Marine  Corps,  the 
Joint-Service  Project,  and  the  research  community.  In  particular,  the  findings  of  an  order 
of  administration  effect  hold  important  implications  for  future  research  in  job 
performance  measurement  throughout  the  domain  of  industrial-organizational  psychology. 
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SUMMARY 


Background 

The  Armed  Services  are  participating  in  a  Congressionally  mandated  project  to  link 
enlistment  standards  to  job  performance.  Consequently,  performance  data  must  be 
gathered,  analyzed,  and  related  to  selection  and  classification  criteria.  Economical 
performance  measures  are  needed  to  complete  this  work.  Therefore,  a  set  of  perfor¬ 
mance  measures  using  different  mezisurement  methods  must  be  developed  and  evaluated. 

Problem 


Each  measure  in  a  set  of  performance  measures  must  be  evaluated  as  an  appropriate 
indicator  of  technical  proficiency.  In  addition,  because  the  order  in  which  the  different 
performance  measures  are  administered  may  vary  for  administrative  and  logistical 
reasons,  sequencing  effects  must  also  be  evaluated. 

Objectives 

Within  the  global  context  of  developing  measures  of  job  performance,  this  study 
focused  on  the  administration  of  multi-method  measures  of  the  individual  job  proficiency 
of  U.S.  Marine  Corps  (USMC)  first-term  3-79  jet  engine  mechanics,  and  the  evaluation  of 
the  performance  measurement  package  on  a  number  of  dimensions.  Specific  objectives 
were  to:  (1)  complete  a  demonstration  of  the  feasibility  of  transferring  testing 
technology  between  services,  (2)  evaluate  the  several  performance  measures  in  terms  of 
user  acceptability,  and  (3)  analyze  effects  of  the  order  of  administration  of  the  hands-on 
and  surrogate  measures. 

Approach 

Four  types  of  performance  measures  were  administered  to  44  USMC  jet  engine 
mechanics  (MOS  6024):  (1)  a  hands-on  job  sample  test;  (2)  an  interview  test;  (3) 

proficiency  ratings  at  the  self,  peer,  and  supervisor  levels;  and  (4)  a  paper-and-pencil  job 
knowledge  test  (JKT).  The  first  three  measures  were  originally  developed  by  the  Air 
Force  for  their  jet  engine  mechanic  specialty  and  subsequently  adapted  for  use  by  the 
naval  services  as  part  of  an  interservice  technology  transfer.  The  fourth  measure,  the 
2KT,  was  developed  by  the  Navy.  The  measures  were  administered  such  that  hedf  of  the 
test  subjects  took  the  3KT  first  and  half,  the  rest  of  the  test  package  first. 

Results 


Order  of  administration  effects  were  found.  Scores  were  affected  by  the  method  of 
measure,  the  tasks  being  evaluated,  and  the  administration  sequence.  There  were  no 
significant  two-way  interactions  between  task  and  sequence.  The  3KT  produced  the 
highest  scores  and  the  interview  component  produced  the  lowest  scores  in  both  the  before 
and  after  administration  groups. 

Of  the  three  types  of  measures  adapted  from  the  Air  Force,  the  hands-on  component 
was  ranked  by  incumbents  as  the  best  instrument  for  providing  accurate  and  useful 
information  about  a  jet  engine  mechanic's  performance.  The  interview  tests  and  rating 
forms  were  ranked  second  and  third  respectively. 


vii 


'd 


Other  user-oriented  information  collected  from  test  subjects  and  supervisors  included 
judgments  of:  (1)  the  comparative  accuracy,  discrimination,  and  acceptability  of  the 
various  ratings  forms;  and  ^2)  the  degree  to  which  job  proficiency  can  be  tapped  by  hands- 
on  tests. 

Conclusions 

1.  The  elements  of  the  Navy/Marine  Corps  JPMS  and  the  2KT  appear  to  exhibit 
sufficient  variance  in  individual  performance. 

2.  Interview  test  items  are  appropriate  for  first-term  jet  engine  mechanics  when 
the  incumbents  have  all  engine  parts  in  view  and  the  task  procedures  are  relatively  short 
and  uncomplicated. 

3.  There  is  evidence  of  significant  order  of  administration  effects  among  different 
parts  of  the  measurement  package. 

4.  The  sample  included  56.5  percent  of  the  total  Marine  Corps  active  duty,  first- 
term  J-79  jet  engine  mechanic  population.  Nevertheless,  the  small  sample  size  militates 
against  strong  generalization  from  the  data  at  hand. 

Recommendations 

1.  The  J-79  performance  measurement  package  should  be  made  available  to 
potential  users  such  as  reserve  components  or  allied  forces  employing  airframes  that 
incorporate  the  J-79  engine. 

2.  Order  of  administration  effects  should  be  taken  into  account  in  all  future 
research  using  multiple  measures  of  performance. 
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INTRODUCTION 


Problem 


Each  measure  in  a  set  of  performance  measures  must  be  evaluated  as  an  appropriate 
indicator  of  technical  proficiency.  In  addition,  because  the  order  in  which  the  different 
performance  measures  are  administered  may  vary  for  administrative  and  logistical 
reasons,  sequencing  effects  must  also  be  evaluated. 

Objectives 


Within  the  global  context  of  developing  measures  of  job  performance,  this  studv 
focused  on  the  administration  of  multi-method  measures  of  the  individual  job  proficiency 
of  Marine  Corps  first-term  J-79  jet  engine  mechanics,  and  the  evaluation  of  the 
performance  measurement  package  on  a  number  of  dimensions.  Specific  objectives  were 
to:  (1)  complete  a  demonstration  of  the  feasibility  of  transferring  testing  technology 
between  services,  (2)  evaluate  the  severed  performance  measures  in  terms  of  user 
acceptability,  and  (3)  analyze  effects  of  the  order  of  administration  of  the  hands-on  and 
surrogate  meeisures. 

Background 

Under  Congressional  mandate,  the  Department  of  Defense  fDoD)  is  conducting  a 
3oint-Service  Job  Performance  Measurement  (JPM)/Enlistment  Standards  Project  aimed 
at  linking  performance  on  the  job  to  the  services'  selection  and  classification  processes. 
In  this  project,  the  services  are  developing  hands-on  performance  measurement  techniques 
for  selected  specialties  unique  to  each  service. 

Job  performance  is  a  complex  phenomenon,  one  of  the  major  aspects  of  which  is 
technical  proficiency.  The  latter  is  the  current  focus  of  Joint-Service  Project  research, 
^or  a  more  extensive  description  of  the  Joint-Service  Project  and  the  Vavy's  7PM 
research  program,  refer  to  Laabs  and  Berry  11987). 

Performance  tests  are  extremelv  time  consuming  and  costly  to  develop.  Con- 
sequentlv,  in  an  attempt  to  effect  fisccd  and  temporal  economies  in  performance 
measurement,  the  services  are  pursuing  a  number  of  cost- reduct  ion  strategies.  One  is  the 
development  of  surrogate  measures.  In  that  effort,  hands-on  tests  are  the  "bench  mark" 
or  standard  against  which  potential  surrogates  (e.g.,  paper-and-pencil  tests,  rating  scales, 
simulations,  etc.)  are  evaluated. 

In  addition  to  investigating  surrogates,  each  service  is  developing  measures  for  at 
least  one  specialty  comparable  across  services,  to  evaluate  the  potential  for  technology 
sharing.  As  part  of  that  effort,  this  study  was  undertaken  to  assess  the  feasibility  of 
transferring  the  measurement  technology  developed  by  one  branch  of  the  armed  forces  to 
another  branch. 

A  jet  engine  mechanic  performance  measurement  package  was  developed  by  the  Air 
Force  (Hedge,  1984)  to  measure  the  performance  of  jet  engine  mechanics  assigned  to  the 
J-79  flight  line  or  shop  maintenance  activities.  The  Air  Porce  test  package  consisted  of  a 
comprehensive  performance  assessment  system  including:  rating  forms  at  the  self,  peer, 
and  supervisor  level  and  a  walk-through  performance  testing  (WTPT)  that  integrated 
hands-on  and  interview  testing.  The  package  was  supplemented  by  questionnaires  eliciting 
background  data,  user  acceptability,  and  so  forth. 
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In  pursuant  to  a  tri-service  cooperative  endeavor,  the  Air  Force  transferred  its 
3-79  test  package  to  the  Navy  for  use  in  evaluating  3-79  jet  engine  mechanics  in  the  naval 
services.  The  deletion  of  the  airframe  employing  the  3-79  engine  resulted  in  there  being 
no  first-term  Navy  jet  engine  mechanics  available  for  testing,  and  only  a  few  such 
personnel  in  the  Marine  Corps.  Nevertheless,  the  data  collection  and  analyses  were 
conducted  in  order  to  gather  all  available  performance  information,  as  well  eis  to  fully 
gauge  the  usefulness  of  the  transferred  test  package. 

The  transfer  itself  went  smoothly  and  proved  both  the  feasibilitv  and  cost 
effectiveness  of  projects  of  that  type.  Details  of  the  technology  transfer  and  the 
development  of  the  resulting  Navv/Marine  Corps  jet  engine  mechanic  test  package  may  be 
found  in  Raker,  Rlackhurst,  and  Alba  0987) 

In  support  of  the  Navy’s  own  investigation  of  potentiad  surrogates  for  hands-on 
testing,  a  paper-and-pencil  job  knowledge  test  (3KT)  was  developed  to  augment  the 
adapted  Air  ^orce  test  package.^  A  descriotion  of  the  procedures  used  to  develop  the 
3KT  is  given  in  Appendix  A. 

For  administrative  and  logistical  reasons,  data  collection  on  the  ^’’M  project  often 
involves  test  subjects  taking  either  the  hands-on  or  a  surrogate  test  first.  The  impact  of 
such  procedures  must  be  addressed  to  ensure  that  performance  is  not  confounded  with 
practice,  which  would  affect  the  findings  of  the  3PM  research  such  as  the  relationship 
between  ASVAB  and  on-the-job  performance.  Because  the  3KT  was  administered  before 
the  WTPT  to  half  the  job  incumbents  and  after  the  WTPT  to  the  other  half,  this  study 
provided  the  opportunity  to  examine  the  effects  of  order  of  administration. 


APPROACH 


Subjects 

All  Marine  Corps  3-79  jet  engine  mechanics  (MOS  602ft)  with  13  to  ft8  months  of 
active  duty  service  and  stationed  in  the  continental  U.S.  or  Hawaii  were  eligible  for 
evaluation  fN  =  65).  However,  factors  such  as  overseas  deployment,  medical  problems, 
and  temporary  duty  assignments  reduced  the  number  available  for  testing,  already  quite 
low  due  to  the  phase  out  of  the  airframe  employing  the  3-79  engine,  to  a  total  of  ftft 
incumbents. 

Data  were  collected  at  four  Marine  Corps  Air  Stations  and  one  Air  Force  Base.  3-7*) 
mechanics  were  drawn  from  both  the  Intermediate-  and  Organizational  Meuntenance 
Activity  (IMA/OMA)  levels,  which  parallel  the  Air  Force  flight  line  and  shop  levels.  The 
sample  included  1  American  Indian,  2  Blacks,  and  ftl  Caucasians.  The  average  length  of 
time  as  a  jet  engine  mechanic  on  the  3-79  engine  was  22  months.  All  incumbents  were 
male.  Table  1  shows  the  number  and  percentage  of  incumbents  by  rank. 


‘Copies  of  the  adapted  Air  Force  test  package  and  the  3KT  may  be  requested  from 
Navy  Personnel  Research  and  Development  Center. 
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Table  1 


Sample  of  3et  Engine  Mechanics 


Rank 

Pay  Grade 

Number 

Percent 

PFC 

E-2 

2 

4 

LCPL 

E-3 

26 

59 

CPL 

E-4 

16 

36 

Measures 

Performance  measures  included  those  transferred  and  adapted  from  the  Air  Force, 
augmented  by  the  3KT.  Table  2  shows  the  list  of  measures;  samples  of  each  measure  are 
contained  in  Appendix  F. 


Table  2 

List  of  Measures 


Type  Title 

Performance  Walk  Through  Performance  Testing  (WTPT) 

•  Hands-on  component 

•  Interview  component 

Overall  Performance  Rating  (OPR) 

Rating  Forms  (comfdeted  by  incumbent,  peer,  and  supervisor) 

•  Global  rating  form 

•  Dimensional  rating  form 

•  T ask  rating  form 

•  Navy/Marine  Corps  wide  rating  form 

3ob  Knowledge  Test  (3KT) 

•  IMA  form 

•  OMA  form 

Auxiliary  Questionnaires 

•  General  utility/acceptability  questionnaire  (GUAO) 

•  background  irrformation  questionnaires 

•  Gener2d  background  questionnaire 

•  Rating  form  questionnaire 

•  Task  experience  rating  questionnaires 
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Walk-through  Performance  Test  (WTPT) 


The  WTPT  is  a  task-level  job  performance  measurement  instrument  incorporating 
both  a  hands-on  component  and  an  interview  component. 

The  hands-on  cc  ponent  was  administered  in  the  work  setting,  being  a  traditional  job 
sample  designed  to  measure  performance  on  a  set  of  critical  job  tasks. 

The  interview  component  was  added  as  a  means  of  addressing  critical  tasks  that 
would  have  been  eliminated  due  to  constraints  such  as  risk  of  personal  injury  or  equipment 
damage.  Interview  testing  also  took  place  in  the  work  setting  and  required  the  evaluator 
to  assess  an  incumbent's  proficiency  on  a  task  by  asking  questions  designed  to  uncover 
knowledge  and  procedural  strengths  and  weaknesses  related  to  the  performance  of  that 
task.  The  incumbent  was  permitted  to  answer  the  questions  by  a  combination  of  verbal 
responses,  gestures,  and  demonstratio'-s. 

Roth  the  WTPT  hands-on  and  interview  test  items  were  administered  from  a  printed 
assignment/score  sheet,  with  yes-no  markings  signifying  correct  or  incorrect  step 
completion.  Point  deductions  were  made  for  step  errors  with  each  step  weighted  by  the 
mean  rating  given  by  subject  matter  experts  of  step  criticality  on  a  9-point  scale. 

An  examinee's  total  WTPT  score  was  obtained  by  summing  the  scores  of  10  hands-on 
and  5  additional  interview  items.  Each  item  was  v^ued  at  10  points;  thus  total  scores 
could  range  from  0  to  150  points.  Figure  I  summarizes  the  WTPT  scoring  procedure. 


Total  WTPT  Score 

•  Range  0  to  150  Points 

•  1  5  Tasks  (10  hands-on,  5  interview! 

•  Total  score  for  an  individual  =  sum  of  all  task  scores 

Task  (Item)  Score) 

•  Range  0  to  10  points 

•  X  number  of  steps 

•  Deductions  made  for  errors  based  on  individual  step  score 

•  Totcil  task  score  =  10  minus  the  total  number  of  points  deducted  for  step 
errors 

Step  Score 

•  Range  varies  depending  on  criticality  and  number  of  steps  in  task 

•  Criticality  rated  by  subject  matter  expert  (SV^F.)  using  9-point  scale  (1  +  low; 
9  =  high) 

•  Establish  step  scores: 

•  Calculate  a  mean  SME  criticality  rating  for  each  step 

•  Sum  the  step  mean  values 

•  Divide  each  step  mean  value  by  the  sum  of  the  mean  values 

•  Multiply  by  10  to  get  error  deductions  per  step 


Figure  1.  Scoring  the  WTPT. 
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Rating  Scales 

Subjectively  derived  overall  performance  ratings  (OPR)  comprised  another  part  of  the 
test  package.  The  ratings  were  assigned  upon  completion  of  every  WTPT  hands-on  and 
interview  task,  but  were  not  included  in  the  total  WTPT  scores.  The  OPRs  were  treated 
as  a  separate,  more  global  method  of  measuring  performance,  to  be  compared  to  the 
WTPT  scores.  That  is,  the  ratings  were  subjective  evaluations  of  overall  performance  and 
did  not  necessarily  reflect  the  number  of  steps  missed  on  a  given  task.  The  evaluators 
considered  such  factors  as  attitude,  speed,  number  of  attempts  to  complete  the  task 
steps,  safety,  as  well  as  successful  task  completion.  The  ratings  were  based  on  a  scale 
from  ]  to  5  as  follows: 

5  Par  exceeded  the  acceptable  level  of  proficiency. 

4  Somewhat  exceeded  the  acceptable  level  of  proficiency. 

3  Met  the  acceptable  level  of  proficiency. 

2  Somewhat  met  the  acceptable  level  of  proficiency. 

1  Was  far  below  the  acceptable  level  of  proficiency. 

Four  rating  forms  completed  the  package  adapted  from  the  Air  Force.  The  following 
ratings  were  completed  at  self,  peer,  and  supervisor  levels:  (I)  the  Global  rating  forms 
(tecfiniccil  and  interpersonal  competence),  (2)  the  Dimensional  rating  form  (occupational 
specialty  behavioral  items'!,  (3)  the  Task  rating  form  (task  specific  measures  (e.g., 
installing  or  removing  components)!,  and  (4)  the  Marine  Corps-wide  rating  form  (military 
related  performance  factors  (such  as  leadership  and  integrity)).  The  rating  scales  were 
behaviorally  anchored  and  ratings  were  on  a  5-point  scale  from  5  (high)  to  1  (low).  Scoring 
was  by  summation. 

Job  Knowledge  Test  (JKT) 

The  Navy-developed  JKT,  a  pa per-and- pencil  simulation,  differed  from  most  paper- 
and-pencil  tests  in  that  it  used  photographs  as  reference  points  and  required  the  examinee 
to:  (1)  identify  components  in  the  photographs,  and  (2)  select  procedures  that  would 
normally  be  followed  when  performing  various  tasks. 

The  JKT  included  10  tasks  or  items  paralleling  the  hands-on  test.  Each  task  had  a 
number  of  "questions"  or  elements  to  which  the  examinee  responded.  Each  task  was  worth 
10  points.  All  questions  were  scored  as  either  correct  or  incorrect.  Total  JKT  score  was 
obtained  by  summing  the  10  task  scores  after  deductions  were  made  for  errors  on  the 
individual  questions.  Error  deduction  points  were  calculated  for  each  task  by  dividing  10 
by  the  total  number  of  questions  in  the  task. 

Examples  of  JKT  scoring  procedures: 

JKT  Task  373  had  15  questions 
T  sk  373  Error  Deduction  Points  =  (10/15) 

■'  ">k  373  Error  Deduction  Points  =  .666 

li  an  examinee  missed  three  questions  on  JKT  task  373,  teisk  score  =  10  - 
(.'--  3)-  1.998  =  8.002. 
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Auxiliary  Measures 

A  General  Utility  Acceptability  Questionnaire  (GUAQ),  also  adapted  from  the  Air 
Force  system,  was  used  to  collect  examinee  evaluation  of  the  fidelity  and  understand- 
ability  of  the  entire  test  package.  Scoring  was  by  summation. 

Three  other  questionnaires  were  also  administered.  These  obtained  background  and 
experience  information  as  well  as  incumbent,  supervisor,  and  peer  opinions  on  the  quality 
of  the  rating  forms.  Scoring  was  by  summation.  Taken  together,  all  of  the  measures 
listed  above  constitute  the  Navy/Marine  Corps  performance  measures  package. 

Test  Items 

Items  contained  in  the  3PMS  are  displayed  in  Figure  ?.  Each  of  these  test  items  was 
addressed  by  one  or  more  of  the  instruments. 


1.  Install  PAcH  valves 

2.  Install  ABFP  valve 

3.  Complete  forms 

4.  Inspect  engine  plumbing 

5.  Inspect  trailer 

6.  Install  bleed  air  system 

7.  Install  anti-icing  duct 

S.  Instcill  EGT  harness 

9.  Install  ignition  exciter  box 

10.  Install  lockwire 

11.  Rig  afterburner  components 

12.  Rig  inlet  guide  vane  systems 

13.  Install  iV3  bearings  (IMA  task) 

14.  Install  //3  oil  seals  (IMA  task) 

15.  Remove  turbine  rotors  (IMA  task) 

16.  Isolate  fuel  malfunctions  (OMA  task) 

17.  Source  of  high  oil  consumption  (OMA  task) 

18.  Install  CSD  assemblies  (OMA  task) 


Figure  2.  Test  items  contained  in  the  JPMS. 

I 
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Hands-on  Tasks 


Items  I,  3,  4,  5,  6,  7,  8,  9,  10,  and  11  shown  in  Figure  2  were  those  of  which 
performance  was  measured  by  hands-on  tasks. 

Common  Test  (Overlap)  Items 


Five  test  items  are  in  common  across  the  hands-on,  the  interview,  and  the  job 
knowledge  tests.  This  provides  for  comparison  of  the  measures  and  the  evaluation  of  the 
JKT  as  a  surrogate  for  hands-on  testing.  These  are  called  "overlap"  items,  figure  3 
graphically  illustrates  the  overlap.  The  remaining  items  are  unique  to  the  particular 
instrument. 


HANDS-ON  TEST 
(10  ITEMS) 


JOB  KNOWLEDGE  TEST 
(10  ITEMS) 


Figure  3.  Overlap  items. 


In  addition,  because  there  are  slight  differences  in  the  duties  of  differently  assigned 
maintenance  personnel,  each  of  the  rating  scales  and  the  3KT  were  developed  in  both  IMA 
and  OMA  forms.  (Refer  to  Baker  et  al.,  1987,  for  further  details.)  Both  3KT  and  the 
rating  scales  were  based  on  the  hands-on  tasks,  and  subject  matter  experts  judged  the  IMA 
and  OMA  tests  to  be  equivalent. 


Procedures 


Rating  forms  and  questionnaires  were  group  administered  to  the  incumbents,  their 
immediate  supervisors,  and  one  co-worker  (peer)  in  a  classroom  setting  on  the  first  day. 
(A  rater  training  session  preceded  rating  form  administration.)  Time  required  was 
approximately  3.5  hours.  Allowing  8  hours  per  subject,  the  WTPT  and  the  JKT  were 
individually  administered  in  the  jet  engine  power  plant  on  the  second  day  and  as  many 
days  thereafter  until  all  subjects  were  evcduated.  The  3KT,  as  noted  earlier,  was 
administered  to  half  the  test  subjects  before  the  WTPT  emd  after  the  WTPT  to  the  other 
half.  The  GUAQ  was  individually  administered  each  day,  after  completion  of  the  WTP'*' 
and  the  3KT.  Total  testing  time  per  individual  was  1  1/2  days.  Background  data  were 
collected  from  individued  service  records. 


RESULTS  AND  DISCUSSION 

Scores  were  obtained  on  all  instruments  for  performance  at  the  task  and  total  score 
levels.  In  addition,  sub-scores  were  generated  on  several  of  the  measures.  However,  the 
small  sample  size  precluded  meaningful  analyses  of  the  relationships  between  the 
measures  at  the  task  level  as  well  as  relationships  between  performance  on  the  various 
measures  and  scores  on  the  ASVAB. 

Multi-method  Performance  Measurement 

For  purposes  of  a  gross  comparison  of  jet  engine  mechanic  job  proficiency  as 
captured  by  the  various  measurement  methods,  only  total  scores  were  used.  These  total 
scores  included:  (1)  WTPT?  (2)  WTPT  hands-on  component;  (3)  WTPT  interview 
component;  (i>)  ratings  by  self  (incumbent),  peer,  and  supervisor  (for  the  10  tasks 
contained  in  the  hands-on  test);  and  (5)  3KT.  Overall  descriptive  statistics  for  results 
attained  on  the  various  measures  are  displayed  in  Table  3. 


Table  3 

Descriptive  Statistics  for  Components  of  the  Multi-method 
Performance  Measurement 


Component/Performance  Levels 

Mean 

SD 

WTPT  (Total) 

106.76 

13.68 

WTPT  (Hands-on) 

75.54 

8.97 

WTPT  (Interview 

65.31 

10.41 

Rating  (Self) 

38.31 

6.22 

Rating  (Peer) 

37.57 

7.37 

Rating  (Supervisor) 

37.47 

7.38 

3KT 

41.21 

3.24 
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Walk- through  Performance  Testing  (WTPT) 

Scores  were  generated  for  each  of  the  10  hands-on  and  10  interview  tasks.  Five  of 
the  interview  tasks  duplicated  or  overlapped  five  of  the  hands-on  tasks;  the  other  five 
tasks  were  different,  depending  on  duty  assignment.  Table  4  displays  descriptive 
statistics  for  IMA  and  OMA  duty  assignments.  Using  only  the  common  hands-on/interview 
items,  the  student's  t-test  reveciled  nonsignificant  differences  between  the  two  groups. 
The  combined  (IMA/OMA)  mean  hands-on  score  for  the  overlap  items  was  38.04,  while  for 
the  five  interview  overlap  items  the  combined  mean  was  34.08. 


T able  4 

WTPT  Total  and  Hands-on  Scores  for  IMA  and  OMA 


Min  ^^ax  Mean  SO 

IMA  (N  =  28) 


WTPT  total 

85.95 

133.86 

106.43 

10.31 

10  hands-on  (Bench  mark) 

66.98 

90.30 

77.79 

5.39 

OMA  (N  =  16) 

WTPT  total 

’7.03 

133.80 

107.35 

18.12 

10  hands-on  (Bench  mark) 

49.47 

90.18 

7)  .59 

12.07 

Overall  Performance  Rating  (OPR) 

Recall  that  at  the  time  of  test  administration,  the  administrator  assigned  an  OPR  to 
the  incumbent's  performance  on  each  of  the  hands-on  and  interview  tasks.  The 
correlation  of  the  OPR  means  with  their  corresponding  hands-on  and  interview  mean 
scores  were  all  positive  and  significant,  ranging  from  a  low  of  .57  for  the  task  Install 
Lockwire  to  a  high  of  .88  for  the  task  Isolate  Fuel  Malfunction. 

Rating  Forms 

Rating  forms  addressing  a  number  of  performance  areas  were  administered  at  the 
incumbent  (self),  peer,  and  supervisor  levels.  The  mean  of  the  Global,  Dimensional, 
Navy/Marine  Corps  wide  and  11  task  mean  ratings  are  reported  for  the  IMA  and  OMA 
combined.  Mean  ratings  over  items  for  each  of  the  four  types  of  ratings  by  each  of  three 
sources  are  shown  in  Table  5.  Means  ranged  on  the  individual  items  from  a  low  of  2.71  to 
a  high  of  4.57. 

Job  Knowledge  Test  (3KT) 

Table  6  displays  the  descriptive  statistics  for  the  JKT  total  score,  the  five 
JKT/inter view/hands-on  overlap  sub-scores,  and  the  five  JKT/interview  non-overlap  sub¬ 
scores  by  activity.  IMA  JKT  total  scores  ranged  from  75.03  to  93.19,  and  OMA  total 
scores  ranged  from  66.61  to  87.50. 
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Table  5 


Grand  Mean  Ratings  Over  Four  Forms  by  Three  Sources 


Source 

Self 

Peer 

Super 

Globa! 

3.92 

3.94 

3.80 

Dimensional 

3.75 

3.71 

3.54 

Task 

3.78 

3.73 

3.73 

Navy/Marine  Corp  wide 

3.58 

3.58 

3.48 

Table  6 


3ob 

Knowledge 

Test  Descriptive 

Statistics 

N 

Min 

Max 

^^eans 

SD 

IMA 

10  OKT  total 

28 

75.03 

93.19 

83.13 

4.54 

5  JKTAnterview/Hands-on 

28 

38.33 

47.67 

42.20 

2.59 

5  3KT /Interview 

28 

34.81 

45.97 

40.93 

2.55 

OMA 

10  OKT  total 

16 

66.61 

87.50 

5  3KT /Interview/Hands-on 

16 

30.89 

44.89 

5  JKT /Interview 

16 

33.15 

42.61 

2.06 

Using  only  the  common  hands-on/interview/3KT  items,  the  students'  t-test  revealed  a 
significant  difference  between  the  two  groups.  Using  only  the  common  (overlap)  tasks, 
the  combined  mean  JKT  score  for  the  overlap  items  was  41.21. 

Results  of  Administration  Sequence 

Tasks  were  evaluated  by  three  methods;  interview,  hands-on,  and  JKT.  The 
interview  component,  which  was  part  of  the  WTPT,  was  always  administered  immediately 
before  the  hands-on  component.  The  3KT,  which  took  approximately  1  hour  to  administer, 
was  administered  before  and  after  WTPT  to  28  and  16  incumbents,  respectively.  Roth 
before  and  after  groups  contedned  IMA  and  OMA  mechanics.  Eighteen  IMA  and  10  OMA 
mechanics  took  the  JKT  before  WTPT;  10  IMA  and  6  OMA  incumbents  took  the  3KT  after 
WTPT. 
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Using  the  data  from  the  five  overlap  items  only,  analyses  of  variance  (AMOVA^  were 
conducted  to  compare:  (1)  the  sequence  of  administration,  and  (2)  the  five  item  scores. 
The  relationship  between  sequence  and  task/item  was  examined  separately  for  each  of  the 
three  methods  used:  interview,  hands-on,  and  3KT.  Tables  7,  and  9  summarize  the 
results  of  these  analyses  for  the  interview,  hands-on,  and  3KT  methods,  respectively. 


Table  7 

Summary  Table:  ANOVA 
(Interview  Method) 


Source 

df 

SS 

MS 

F 

P 

Main  Effects 

5 

188.5 

37.7 

13.9 

<.01 

Task 

4 

156.5 

39.) 

14.4 

<.0! 

Sequence 

1 

32.0 

32.0 

11.8 

<.01 

Interaction 

Task/Sequence 

4 

10.2 

2.5 

.9 

.tilt 

Explained 

o 

198.7 

22.0 

8.1 

<.0! 

Residual 

210 

568.1 

2.7 

— 

— 

Total 

219 

766.9 

3.5 

— 

— 

Table  8 

Summary  Table;  ANOVA 

(Hands-on) 

Source 

df 

SS 

MS 

F 

P 

Meiin  Effects 

5 

369.0 

73.8 

19.1 

<.0i 

Task 

4 

349.1 

87.2 

22.6 

<.0I 

Sequence 

1 

19.9 

19.9 

5.1 

.02 

Interaction 

Task/Sequence 

4 

11.8 

2.9 

.7 

.54 

Explained 

9 

380.8 

42.3 

10.9 

<.01 

Residual 

210 

809.6 

3.8 

— 

— 

Total 

219 

1190.5 

5.4 

— 

— 
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Table  9 

*;ummary  Table;  ANOVA 
(Job  Knowledge) 


Source 

df 

SS 

MS 

F 

P 

Main  Effects 

5 

134.2 

26.8 

16.7 

<.ni 

Task 

4 

123.5 

30.8 

19.3 

<.oi 

Sequence 

1 

10.6 

10.6 

6.6 

.0! 

Interaction 

Task/Sequence 

4 

7.3 

1.8 

1.1 

.33 

Explained 

9 

141.6 

15.7 

9.8 

<.0I 

Residual 

210 

335.9 

1.6 

— 

— 

Total 

219 

477.5 

2.1 

— 

— 

Main  Effects 

Review  of  Tables  7 

through 

9  indicates  significant  differences  among 

tasks  and 

between  sequence  for  each  of  the  three  methods. 

Task  and  sequence  means 

by  method 

are  displayed  in  Table  10. 

Table  10 

Task  and  Sequence  by  Method 

Interview 

Hands-on 

Knowledge 

Task  Before 

After 

Before 

After 

Before 

After 

R1  6.7 

5.2 

7.1 

5.8 

8.2 

8.7 

134  8.0 

7.7 

6.2 

6.1 

6.4 

7.4 

353  8.0 

7.2 

8.9 

8.2 

8.2 

8.8 

360  6.2 

5.3 

7.2 

6.3 

8.3 

8.8 

373  6.4 

6.0 

9.5 

9.4 

9.0 

8.8 

Mean  7.1 

6.3 

7.8 

7.1 

8.0 

8.5 

The  sequence  in  which  the  knowledge  test  was  administered  (before  or  after 
interview/hands-on)  appears  to  have  an  effect  on  examinee's  scores  on  both  the  hands-on 
and  interview  portions  of  the  WTPT.  On  the  other  hand,  the  JKT  was  also  affected  by  the 
testing  sequence,  in  that  mean  knowledge  scores  for  the  "after"  group  are  higher  than  the 
mean  knowledge  scores  for  the  "before"  group.  In  group  comparisons,  these  effects  would 
likely  cancel  each  other,  which  is  the  intended  effect  of  counterbalancing,  ^'hen  test 
performance  data  is  to  be  related  to  a  set  of  predictors,  however,  the  individual  scores 
must  be  on  a  common  metric.  In  the  present  case,  rank  ordering  of  individuals  may 
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change  depending  on  what  instrument  among  the  interview,  hands-on  test,  and  job 
knowledge  was  administered  first. 


Although  it  is  impossible  to  assess  with  the  present  design,  hands-on  test  scores  may 
have  been  elevated  by  exposure  to  "interview"  test  items  in  the  WTPT  in  addition  to  the 
job  knowledge  test  items. 

Thus,  the  problem  of  sequence  effects  is  exacerbated  in  a  multi-method  performance 
measurement  package.  Our  results  suggest  that  multiple  exposures  to  similar  subject 
matter  regardless  of  measurement  method  will  tend  to  increase  examinee  scores  and  this 
must  be  taken  into  account  in  future  performance  measurement  efforts. 

Questionnaires 

Incumbents  responded  to  the  rating  form,  experience  (as  a  3-79  mechanic  and  on 
particular  tasks),  background,  and  acceptability  questionnaires.  Peers  and  supervisors 
responded  to  the  rating  form  and  background  questionnaires. 

Experience  Questionnaire 

IMA  personnel  indicated  they  had  been  jet  engine  mechanics  for  an  average  of  26 
months  with  an  average  of  24  months  in  the  IMA  shop.  OMA  mechanics  averaged  20 
months  as  jet  engine  mechanics  and  15  months  in  the  OMA  shop.  The  IMA  group  reported 
more  experience  than  the  OMA  group  for  10  of  the  12  common  IMA/OMA  tasks,  less 
experience  for  the  remaining  two  tasks.  Table  1 1  shows  the  mean  experience  reported  by 
each  group  for  each  of  the  12  common  IMA/OMA  tasks.  The  difference  in  experience  may 
explain  the  better  performance  of  the  IMA  groups  previously  illustrated  in  Tables  4  and  6. 


Table  1 1 

IMA  and  OMA  Mean  ''"ask  Experience  for  the  12  Common  IMA/OMA  Tasks 

(IMA  N  =  28,  OMA  N'  =  16) 


Task 

No. 

Task  Description 

Type 

Mean  Experience 
IMA  OMA 

R1 

Install  P&D  valve 

Hands-on 

4.07 

3.38 

R2 

Install  ABFP  valve 

Interview 

4.46 

3.25 

134 

Complete  VIDS  MAP 

Hands-on 

5725 

4.56 

301 

Inspect  engine  plumbing 

Hands-on 

5.21 

4.06 

302 

Inspect  trailer 

Hands-on 

3.21 

4.31 

349 

Install  bleed  air  comp 

Hands-on 

5.50 

3.75 

353 

Install  anti-icing  duct 

Hands-on 

4,61 

2.56 

360 

Insteill  EGT  harness 

Hands-on 

4.71 

2.69 

363 

Install  exciter  box 

Hands-on 

3.64 

2.88 

373 

Install  locl<wire 

Hands-on 

6.75 

5.88 

385 

Rig  afterburner  comp 

Hands-on 

02 

3.44 

387 

Rig  inlet  guide  vane  system 

Interview 

3.50 

2.75 

Note.  Underline  indicates  more  experience;  Sczde:  1  =  No  Experience,  7  =  A  Very  Great 
Amount  of  Experience. 
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Rating  Form  Questionnaire 

Incumbents,  peers,  and  supervisors  were  asked  to  rate  and  rank  the  four  rating  forms 
for  accuracy,  discrimination,  and  acceptability,  and  to  indicate  how  motivated  they  were 
to  complete  the  rating  forms.  Table  12  shows  the  self,  peer,  and  suoervisor  rating  form 
questionnaire  mean  ratings  and  rankings  of  the  four  different  forms. 


Table  12 

Rating  Form  Questionnaire  Mean  Ratings  and  Rankings  By  Source 

(N  =  44) 


Ratings 

(1  =  Low;  5  =  High) 

Self 

Source 

Peer 

Supervisor 

How  motivated 

3.30 

3.05 

3.82 

Accuracy  of  Rating  Form; 

Global 

3.77 

3.30 

3.73 

Dimensional 

3.73 

3.41 

4.07 

Task 

3.80 

3.36 

3.89 

Navy/ Marine  Corps 

3.55 

3.27 

3.70 

Discrimination  of  Rating  Form: 

Global 

3.39 

3.07 

3.25 

Dimensional 

3.55 

3.25 

3.57 

Task 

3.61 

3.34 

3.61 

Navy  Marine  Corps 

3.23 

3.02 

3.50 

Acceptability  of  Rating  Form: 

Global 

3.30 

7.93 

3.14 

Dimensional 

3.43 

3.11 

3.59 

Task 

3.45 

3.14 

3.64 

Navy/Marine  Corps 

3.16 

2.95 

3.34 

Accuracy  of  Rating  Form; 

Global 

3.0  =  4 

3.0  = 

4 

3.0  =  4 

Dimensional 

2.4  =  2 

2.1  = 

2 

2.2  =  2 

Task 

1.8  =  1 

1.7  = 

1 

2.0  =  1 

Navy/Marine  Corps 

2.9  =  3 

2.3  = 

3 

2.6  =  3 

Discrimination  of  Rating  Form; 

Global 

2.8  =  4 

2.9  = 

4 

2.8  =  4 

Dimensional 

2.7  =  3 

2.2  = 

2 

2.4  =  2 

Task 

1.9  =  1 

1.8  = 

1 

1.9  =  1 

Navy/Marine  Corps 

2.6  =  2 

2.2  = 

3 

2.6  =  3 

Acceptability  of  Rating  Form; 

Global 

3.0  =  4 

3.0  = 

4 

3.1  =  4 

Dimensional 

2.6  =  2 

2.1  = 

2 

2.1  =  2 

Task 

1.7  =  1 

1.6  = 

1 

1.9  =  1 

Navy/Marine  Corps 

2.8  =  3 

2.5  = 

3 

2.7  =  3 
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Motivation.  All  three  sources  indicated  that  they  were  motivated  to  comolete  the 
rating  forms,  with  supervisors  indicating  the  highest  motivation.  Peers,  who,  interestingly 
enough,  turned  out  to  be  the  best  oredictors  of  the  incumbents'  test  performance, 
reported  the  least  motivation  to  complete  the  rating  forms. 

Discrimination.  All  raters  believed  that  they  could  discriminate  between  good  and 
poor  performers  by  responses  to  any  one  of  the  four  rating  forms.  All  sources  gave  a 
"moderate"  to  "great  extent"  discrimination  rating  for  all  four  rating  forms. 

Accuracy.  All  three  sources  gave  a  "moderate"  to  "great  extent"  rating  for  accuracy 
of  the  Global,  Task,  and  Meirine  Corps-wide  rating  forms.  The  supervisor  group  responded 
with  ratings  of  either  "to  a  great  extent"  or  "to  very  great  extent"  accuracy  rating  for  the 
Dimensional  rating  form,  while  self  and  peer  raters  gave  that  form  a  "moderate"  to  "great 
extent"  accuracy  rating. 

Acceptability.  All  but  one  acceptability  rating  fell  between  "moderate"  and  to  "a 
great  extent."  The  peer  raters  gave  a  "small"  to  "moderate"  rating  for  acceptability  of 
the  Global  and  Navy/Marine  Corps-wide  rating  forms. 

Rankings.  All  three  sources  ranked  the  Task  rating  form  as  the  best  for  accuracy, 
discrimination,  and  acceptability,  and  the  Navy/Marine  Corps-wide  rating  form  third  for 
accuracy  and  acceptability.  Incumbents  (self)  ranked  the  Navy/Marine  Corps-wide  rating 
form  and  the  Dimensional  rating  form  second  and  third  respectively  for  discrimination, 
with  peers  and  supervisors  ranking  the  two  forms  in  reverse  order.  All  three  sources 
ranked  the  Global  rating  form  fourth  in  accuracy,  discrimination,  and  acceptability. 

General  Utility/Acceptability  Questionnaire 

Incumbents  were  asked  to  rate  the  usefulness  of  the  3PM  System  components  in 
terms  of  accuracy  and  performance  discrimination.  Table  13  shows  the  mean  responses. 

Incumbents  indicated  that  they  believed  that  job  proficiency  could  be  determined  by 
the  hands-on  component  to  a  "very  great  extent,"  by  the  rating  forms  to  a  "small  extent," 
and  by  the  interview  to  a  "moderate  extent."  Incumbents  believed  that  the  hands-on  tests 
were  acceptable  to  a  "great  extent"  and  that  the  interview  and  rating  forms  were 
acceptable  to  a  "moderate  extent." 

The  incumbents  indicated  that  they  were  most  motivated  to  complete  the  hands-on 
test  and  leeist  motivated  to  complete  the  rating  forms.  Incumbents  indicated  that  they 
were  prepared  from  a  "moderate"  to  "great  extent"  to  complete  the  hands-on,  interview, 
and  rating  forms.  They  believed  the  importance  of  the  program  was  expressed  from  a 
"moderate"  to  "great  extent."  The  incumbents  were  concerned  to  a  "smsill  extent"  that 
the  information  being  collected  might  be  used  for  other  purposes. 

Incumbents  ranked  the  hands-on  component  best  for  ability  to  provide  accurate  and 
useful  information  about  a  jet  engine  mechanic’s  performance.  The  interview  tests  and 
rating  forms  were  ranked  second  and  third  respectively. 


Table  13 

General  Utility/ Acceptability  Questionnaire 
Incumbent  Mean  Ratings  and  Rankings 
(N  =  UU) 


Rating 

Ranking 

Belief  in: 

Rating  forms 

2.77 

Hands-on  tests 

4.07 

Interview  tests 

3.07 

Acceptability  of: 

Rating  forms 

3.02 

Hands-on  tests 

4.07 

Interview  tests 

3.32 

Motivation  to  complete: 

Rating  forms 

2. go 

Hands-on  tests 

3.g4 

Interview  tests 

3.23 

Preparation  for: 

Rating  forms 

3.55 

Hands-on  tests 

3.95 

Interview  tests 

3.g2 

Importance  expressed 

3.80 

Concerned 

2.93 

Rating  forms 

3.48 

3 

Hands-on  tests 

1.25 

1 

Interview  tests 

2.68 

2 

Note.  Scales;  Rating  (1  =  Low;  5 

=  High),  Ranking  (1  =  Best;  3  =  Worst). 

Background  Questionnaire 

Table  14  shows  the  incumbent,  peer,  and  supervisor  opinions  regarding  the  situational 
constraints  in  the  work  environment,  motivation  factors,  and  commitment.  All  three 
sources  agreed  that  the  technical  manuals  were  understandable  and  available  and  that  the 
jet  engine  mechanics  were  able  to  use  their  skills  on  the  job.  A  sense  of  accomplishment, 
feeling  that  their  jobs  were  important,  arid  a  belief  that  they  performed  their  duties  to 
the  best  of  their  ability  were  evidenced  by  the  members  of  the  three  groups.  The  three 
sources  believed  that  on-the-job  training  (OJT)  provided  the  skills  required  to  perform 
their  jobs. 

The  three  groups  differed  in  their  beliefs  about  tools  being  available  when  needed. 
Both  incumbents  and  peers  believed  that  their  supervisors  were  concerned  about  them  and 
that  the  supervisors  gave  them  the  support  they  required;  the  supervisors  only  slightly 
agreed  on  both  issues.  The  supervisors  agreed  that  they  had  self  pride,  job  satisfaction, 
and  a  sense  of  responsibility,  but  peers  sind  incumbents  only  slightly  agreed  that  they 
possessed  these  three  motivational  factors.  The  supervisors  agreed  that  the  technical 
schools  provided  them  with  the  skills  that  they  needed  for  the  job,  but  the  incumbents  and 
peers  were  more  equivocal. 
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Table  lU 


Work  Environment,  Motivation,  and  Commitment  Mean  Ratings 

(N  =  44) 


Item 

Self 

Source 

Peer 

Supervisor 

Manuals  understandable 

4.09 

3.85 

4.18 

Manuals  available 

4.16 

4.27 

4.25 

Tools  available 

3.25 

3.34 

3.32 

Able  to  use  skills 

4.18 

4.15 

4.66 

Sense  of  accomplishment 

4.16 

3.98 

4.14 

Supervisor  concerned 

3.98 

3.88 

3.43 

Supervisor  gives  support 

4.00 

3.90 

3.66 

!Job  interesting 

3.95 

3.93 

4.61 

Self  pride 

3.57 

3.68 

4.50 

Job  importance 

4.09 

4.05 

4.82 

Job  satisfaction 

3.73 

3.71 

4.18 

Sense  of  responsibility 

3.45 

3.46 

4.55 

Perform  duties  to  best  of  ability 

4.36 

4.24 

4.75 

Technical  schools  provided  skills 

3.32 

3.27 

4.70 

On-the-job  training  provides  skills 

4.34 

4.20 

4.64 

Note.  Scale;  1  =  Strongly  Disagree,  5  =  Strongly  Agree. 


Other  Relationships 

In  the  !Ioint-Service  research  strategy  (Laabs  &  Eerry,  19S7),  hands-on  tests  are  the 
bench  mark  to  which  potential  surrogates  for  hands-on  tests  must  be  compared.  This,  of 
course,  may  be  done  at  the  global  or  item  levels,  or  both.  The  comparisoii  of  ootential 
surrogates  with  each  other  may  also  prove  useful.  Unfortunately,  given  the  sample  size 
and  the  pronounced  order  of  administration  effects,  this  was  not  possible  in  the  present 
study. 

Again,  given  the  sample  size  and  the  pronounced  order  of  administration  effects,  it 
was  not  possible  to  correlate  JPMS  components  with  selection  and  classification  factors 
such  as  incumbent  background  information  fpay  grade,  high  school  completion  (diploma, 
GED),  3-79  school  final  course  grade),  or  scores  attained  on  the  ASVAA. 

Discussion 

In  this  data  collection  on  the  performance  of  44  Marine  Corps  3-79  jet  engine 
mechanics,  personnel  were  evaluated  by  means  of  a  multi-method  testing  system;  hands- 
on,  interview,  psper-and-pencil  job  knowledge,  measures  and  performance  ratings.  The 
hands-on  tests  should  be  considered  to  be  sound  measures  of  performance  since  their 
content  was  directly  linked  to  the  step-by-step  procedures  required  to  perform  each  task. 
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The  interview  component  appeared  to  have  a  stronger  relationship  than  the  rating 
form  component  with  the  hands-on  component.  Thus,  the  interview  component  may  be  a 
better  discriminator  of  hands-on  performance  than  the  rating  form  component. 

A  possible  explanation  for  the  strange  relationship  between  interview  and  hands-on 
may  be  related  to  the  content  and  design  of  the  interview  items  and  to  the  amount  of 
experience  reported.  Two  of  the  OMA  unique  interview  items  required  the  incumbents  to 
answer  a  series  of  questions  related  to  two  troubleshooting  tasks.  The  procedures  were 
similar  to  those  actually  performed  on  the  job. 

The  other  OMA  unique  item  and  seven  IMA/OMA  common  interview  items  required 
the  incumbents  to  recall  and  explcun  procedures  performed  on  the  engine's  external  parts. 
The  incumbents  could  see  and  point  to  the  appropriate  engine  parts  while  explaining  the 
procedures  of  the  nine  tasks.  All  three  of  the  IMA  unique  interview  items  required  the 
incumbents  to  reccdl  and  explain  complicated  procedures  performed  on  the  engine's 
internal  parts.  The  IMA  incumbents  were  unable  to  see  the  parts  and  thus  use  them  as 
reference  points  when  attempting  to  explain  the  three  IMA  procedures. 

Many  incumbents,  even  some  who  had  performed  these  tasks  several  times,  were 
unable  to  respond  correctly  to  the  "show  and  tell"  requirement.  This  did  not  necessarily 
mean  that  the  mechanics  were  unable  to  read  the  publications  and  perform  the  hands-on 
tasks  at  an  acceptable  or  better  level  of  performance. 

This  leads  to  the  belief  that  "show  and  tell"  interview  test  items  are  appropriate  for 
first-term  jet  engine  mechanics  when  the  incumbents  have  all  engine  parts  in  view  and  the 
task  procedures  are  relatively  short  and  uncomplicated.  When  the  parts  are  hidden  from 
the  incumbent's  view  and  the  procedures  are  relatively  long  and  complicated,  some  other 
method  of  evaluation  should  be  developed  or  the  task  should  be  excluded  from  the 
evaluation  procedure.  One  alternative  would  be  to  provide  the  subjects  with  pictures  or 
drawings  for  reference  points  of  the  unobservable  areas.  (A  more  thorough  evaluation  of 
interview  tests  as  surrogates  for  hands-on  performance  tests  is  being  conducted  by  the  Air 
Force.l 

It  is  appropriate  to  note  that  the  order  in  which  the  JKT  and  the  hands-on/interview 
testing  were  administered  was  found  to  have  definite  effects.  Scores  were  affected  by 
the  method  of  administration,  the  tasks  being  evaluated,  and  the  JKT/WTPT 
administration  sequence.  The  JKT  produced  the  highest  scores  and  the  interview 
component  produced  the  lowest  scores  regardless  of  the  order  of  administration.  There  is 
no  apparent  reason  for  the  significant  difference  in  performance  on  the  3KT  between  IMA 
and  OMA  personnel. 

Of  more  far-reaching  import  are  the  questions  of  what  is  an  acceptable  surrogate  for 
a  hands-on  (job  sample)  test  (a  question  that  must  be  decided  at  the  Joint-Service  level), 
and  the  relationship  of  the  various  measures  to  the  ASVAB. 

The  sample  of  44  included  56.5  percent  of  the  toted  Marine  Corps  active  duty,  first- 
term  3-79  jet  engine  mechanic  population.  This  small  sample  size  militates  against  strong 
generalization  about  the  overall  proficiency  of  jet  engine  mechanics  or  the  relationship 
between  ASVAB  and  03T  performance. 

In  the  end,  because  of  the  small  sample  size,  the  most  significant  findings  in  this 
effort  (besides  the  demonstration  of  interservice  transferability  of  the  performance 
measures)  were  the  indications  of  order-of-administration  effects.  Because  all  of  the 
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services  are  using  surrogates  along  with  hands-on  tests,  which  may  be  administered  in 
different  sequences,  practice  effects  should  be  controlled  for. 


CONCLUSIONS 

1.  The  elements  of  the  Navy/Marine  Corps  JPMS  and  the  3KT  appear  to  exhibit 
sufficient  variance  in  individual  performance. 

2.  Interview  test  items  are  appropriate  for  first-term  jet  engine  mechanics  when 
the  incumbents  have  all  engine  parts  in  view  and  the  task  procedures  are  relatively  short 
and  uncomplicated. 

3.  There  is  evidence  of  significant  order  of  administration  effects  among  different 
parts  of  the  measurement  package. 

The  sample  included  56.5  percent  of  the  total  Marine  Corps  active  duty,  first- 
term  3-79  jet  engine  mechanic  population.  Nevertheless,  the  small  sample  size  militates 
against  strong  generalization  from  the  data  at  hand. 


RECOMMENDATIONS 

Because  no  active  duty  sample  exists  on  which  to  conduct  further  research  with  these 
instruments,  research  on  the  3PMS  for  3-7^  jet  engine  mechanics  should  be  terminated. 
The  3-79  3PMS  should  be  made  available  to  potential  users  such  as  reserve  components  or 
allied  forces  employing  airframes  that  incorporate  the  3-79  engine.  In  one  specific 
instance,  copies  of  the  test  materials  were  forwarded  to  Headquarters,  4th  Marine  Air 
Wing  (FMF),  U.S.  Marine  Corps  Reserve  for  possible  use  in  USMCR  3-79  jet  engine 
mechanic  performance  measurement  and  skills  certification.  Future  Navy  3PM  research 
should  control  for  order  of  administration  effects.  Also,  a  study  should  examine  whether 
certcun  tasks  are  better  addressed  by  different  types  of  measures. 
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APPENDIX  A 

30A  KNOWLEDGE  TEST  DEVELOPMENT 


A-0 


30B  KNOWLEDGF  TEST  DEVELOPMENT 


Test  development  usually  requires  preliminary  work  to  select  tasks  that  are 
representative  of  the  job  domain  to  be  evaluated  and  organize  those  tasks  into  testable 
steps  (or  sub-tasks).  For  the  development  of  the  3et  Engine  Mechanic  3KT,  however, 
neither  of  these  efforts  were  required  because  the  tasks  were  selected  and  analyzed 
during  the  development  of  the  interview  components  of  the  Navy  TPMS.  (See  Baker, 
Blackhurst,  &  Alba,  1987  for  details  of  the  TPMS.)  Two  forms  of  the  TKT  were  developed, 
one  for  the  intermediate  maintenance  activity  (IMA)  level  and  one  for  the  organizational 
maintenance  activity  (OMA)  level.  Table  A-1  lists  the  tasks  that  compose  the  two  forms 
of  the  3KT. 


Table  A-1 

Tasks  Included  in  Navy/Marine  Corps  Tob  Knowledge  Test 
(IMA  and  OMA  Forms) 


Task 

Task  No. 

IMA 

OMA 

Instcill  lockwire 

3373 

X 

X 

Install  pressurizing  and  drain  valves 

3  R) 

X 

X 

Install  afterburner  fuel  pressurizing  valves 

3  R2 

X 

X 

Install  anti-icing  ducts 

3353 

X 

X 

Install  Egt  thermocouple  harnesses 

3360 

X 

X 

Rig  inlet  guide  vane  system 

3387 

X 

X 

Complete  forms 

3134 

X 

X 

Install  constant  speed  drives 

3351 

X 

Isolate  fuel  system  malfunctions 

3319 

X 

Determine  source  of  high  oil  consumption 

3325 

X 

Install  number  3  bearings 

3238 

X 

Install  number  3  oil  seals 

3239 

X 

Remove  turbine  assemblies 

3247 

X 

In  order  to  identify  all  of  the  appropriate  steps  for  each  of  the  tasks,  the  test 
developer  and  two  SMEs  assigned  as  treuners  at  Marine  Corps  Air  Station  Beaufort,  South 
Carolina,  reviewed  the  relevant  Navy  technical  publications.  False  steps  were  cilso 
identified  and  served  as  step  distractors  for  each  task.  Test  items  were  written  for  both 
the  IMA  and  OMA  forms.  Each  form  included  10  tasks.  Seven  items  were  common  to 
both,  and  three  items  were  unique. 

Photographs  for  the  TKT  were  taken  at  Naval  Air  Station,  Dallas.  The  photographs 
depicted  a  J-79  jet  engine,  individual  engine  components,  and  the  various  materials  used 
to  remove  or  replace  parts  and  sections  of  the  engine.  The  components  were  photo¬ 
graphed  on  and  off  the  engine  to  allow  for  flexibility  when  developing  the  3KT  items. 
Sizing  of  the  photographs  was  conducted  on  site  by  the  photographer,  under  the  direction 
of  an  active  duty  jet  engine  mechanic  SME  and  the  test  developer.  Each  photograph 
contained  the  appropriate  component  of  the  engine  that  the  examinee  was  to  identify 
during  the  testing  situation,  2ts  well  as  at  least  two  other  components  that  served  as  test 
distractors  for  each  task. 
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The  tests  items  were  arranged  in  a  test  book  with  the  photograph(s)  on  the  left  page 
and  the  steps  and  distractors  on  the  page  to  the  right,  so  the  examinee  could  view  the 
photographs  and  respond  to  the  questions  without  being  required  to  turn  the  page.  A  short 
verbal  scenario  weis  provided  at  the  top  of  each  right-hand  page  (see  Figure  AF-1)  to 
accompany  the  respective  photo  reference  point  and  to  set  the  stage  for  each  task. 
Instructions  indicated  that  the  examinees  were  to  record  their  responses  on  an  answer 
sheet  provided  by  the  test  administrator. 


SCENARIO  FOR  TASK  3R1:  You  have  been  instructed  to  install  a  pressurizing  and  drain 
(PdcD)  valve  on  the  engine.  The  main  oil  cooler,  compressor  rear  frame  bracket  and 
lines  are  ad  ready  on  the  engine. 

3.  (1016)  Which  component  on  the  picture  to  the  left  is  the  pressurizing  and  drain  fPicH) 
valve?  Write  the  matching  letter  on  the  answer  sheet. 

4.  From  the  list  below,  select  the  actions  or  checks  that  you  should  take  when  installing 
the  P&D  valve.  Place  a  check  mark  in  the  yes  column  of  the  answer  sheet  if  you  should 
perform  the  action.  Place  a  check  mark  in  the  no  column  if  you  should  not  perform  the 
action. 

(1017)  Lubricate  the  0-rings  and  seal  with  graphite  grease  prior  to  installation. 

^1018)  Install  elbows,  jam  nuts,  and  O-rings  on  the  large  outlet  and  rear  ports. 

(1019)  Leave  the  jam  nuts  finger  tight  until  after  the  fuel  manifolds  are  installed. 

(1070)  Instcdl  a  clamp  bracket  between  the  PicD  valve  and  the  main  oil  cooler. 

(I0?1)  Install  the  valve  in  the  correct  position  with  the  ports  facing  the 

appropriate  lines. 

(1022)  Torque  the  four  bolts  holding  the  valve  to  the  main  oil  cooler. 

(1023)  Safety  wire  all  four  bolts  together  in  one  series. 

(1024)  Install  two  bolts  to  secure  the  valve  to  the  rear  mounting  bracket. 

(1025)  Instcdl  an  O-ring,  drain  tube,  O-ring,  and  connector  bolt  (in  that  order)  in 

the  drain  port. 

(1026)  Torque  the  two  bolts  securing  the  valve  to  the  rear  mounting  bracket. 

(1027)  Position  the  two  large  elbows  to  the  rear  and  use  a  common  screwdriver  to 

align  the  fittings  when  installing  the  fuel  manifolds. 

(1028)  Torque  the  jam  nuts  and  manifolds,  and  lockwire. 

(1029)  Position  the  rear  elbow  and  install  the  reference  fuel  pressure  manifold 
tube. 

(1030)  Torque  the  jam  nut  and  tube  coupling  and  lockwire. 


Figure  AF-1.  Example  of  job  knowledge  test. 
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U.S.  Marine  Corps  SMEs  reviewed  the  completed  JKT  for  accuracy  and  understand- 
ability.  A  pilot  test  was  conducted  at  Naval  Air  Station,  Dallas  by  experienced  test 
administrators  who  had  received  extensive  training  in  test  administration  and  had 
previously  served  in  analogous  testing  of  3-79  jet  engine  mechanics  for  an  Air  Force 
project. 

The  test  was  administered  to  three  first-term  jet  engine  mechanics,  one  from  the 
organizational  activity  and  two  assigned  to  the  shop  maintenance  areas.  The  test  subjects 
were  allowed  as  much  time  as  necessary  to  complete  the  test.  All  three  jet  engine 
mechanics  completed  the  3KT  within  1  hour. 

Subsequent  to  the  pilot  test,  minor  editorial  changes  were  made  and  the  3KT  was 
printed  and  compiled  for  administration  to  jet  engine  mechanics  assigned  to  USMC 
intermediate  and  organizational  maintenance  activities. 
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APPENOIX  B 

SAMPLE  PAGES  OF  30P  PERFORMANCE  MEASUREMENT  SYSTEM 


B~0 


GENERAL  BACKGROUND 


YOUR 

NAME  _  SSN 


The  following  questions  pertain  to  your  work  experience,  your  work  unit,  and 
your  feelings  about  your  job.  This  Information  will  be  used  for  research 
purposes  only.  Please  check/fill  In  each  blank  as  accurately  as  possible. 


1 .  Present  Rank/Rate : 

E-1 

E-5 

E-2 

E-6 

E-3 

E-7 

E-4 

E-8 

2.  Months  In  present  unit:  _ 

3.  Months  you  have  been  a  Jet  Engine  Mechanic  on  the  J-79  engine  model  system 

4.  Area  where  most  of  your  work  is  done:  _  IMA  _  OMA 

5.  Months  In  IMA: _ ;  months  in  OMA: _ 

6.  Please  list  any  additional  Jet  Engine  Mechanic  experience  below. 

Engine  Type  Amount  of  Experience  (months) 


7.  In  general,  how  is  morale  in  your  unit?  (Check  one) 

_  Extremely  high 

_  Fairly  high 

_  Average 

_  Fairly  low 

_  Extremely  low 


TURN  SHEET  OVER  FOR  ADDITIONAL  QUESTIONS 
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Phase  I  J-79 

IMA  and  OMA 


Hands-On  Task  349 


Objective;  To  evaluate  the  incumbent's  ability  to  install  J-79 
engine  bleed  air  system  components  (manifold  collector 
bowls) . 

Estimated  Time;  30M  Start; _ Finish: _ Time  Reg; _ 

Time  Limit;  45M  #Times  Performed; _ t^jist  Perform?^? _ 

Tools  and  Equipment;  PUB;  NAVAIR  01-245FDD-2-3-5 ,  Pg  3-46,  Tool 
Box,  Petrolatum  or  Tape,  0-150  inch-pound  Torque  Wrench, 
0-300  inch-pound  Torque  Wrench,  one  50887-550S  Conoseal,  one 
MAO  06-005  Gasket. 

Background  Information;  Evaluate  on  modified  bleed  air  system. 

Engine  Configuration;  The  modified  BLC  system  with  the  BLC 
ducts,  the  BLC  support,  and  the  short  duct  are  installed  on 
the  engine.  The  cap  and  attachment  hardware  are  removed 
from  the  manifold  collector  bowl.  The  manifold  collector 
bowl  is  off  the  engine.  Do  NOT  remove  the  bracket  assembly 
from  the  engine  (remove  six  o'clock  bolt) . 

Instructions; 

Administer  in  the  shop.  The  incumbent  may  use  the  PUB.  Hold  the 
rig  pin  in  while  the  incumbent  tightens  the  rear  end  jam  nuts. 
Show  the  incumbent  the  manifold  collector  bowl  to  ensure  that 
he/she  understands  what  component  is  to  be  installed. 

SAY  TO  THE  INCUMBENT 

INSTALL  THE  BLEED  AIR  SYSTEM  MANIFOLD  COLLECTOR  BOWL  USING  PUB 
NAVAIR  01-245FDD-2-3-5,  PG  3-46  AS  A  GUIDE.  FOLLOW  GENERAL 
MAINTENANCE  PROCEDURES  AT  ALL  TIMES.  TELL  ME  IF  YOU  PLAN  TO 
DEVIATE  FROM  THE  PUB. 

Performed  or  Answered  Correctly  Yes  No 

Did  the  incumbent: 

1.  Ensure  that  the  conoseal  gasket  was 
installed  with  the  gasket  angle 

oriented  with  the  flange  face  of  the  duct?  _  _ 

2.  Position  the  collector  coupling  (V-Band  clamp) 
split  line  at  the  3  or  9  o'clock  position 

with  the  bolt  facing  down?  _  _ 
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Interview  Task  Ri 


Phase  I  J-79 

IMA  and  OMA 


objective;  To  evaluate  the  incumbent's  knowledge  concerning  the 
installation  of  pressurizing  and  drain  (P&D)  valves. 

Estimated  Time;  lOM _ Start;  Finish;  Time  Rea; _ 

Time  Limit;  12M _ #Times  Performed; _ Last  Performed; _ 

Tools  and  Equipment;  PUB;  NAVAIR  02B-105AGD-6-1  WP  119  00,  P&D 
Valve,  Tool  Box. 

PaoHqrgpnd  inforroatidn:  n/a 

Engine  Configuration;  N/A 
Instructions; 

Administer  in  the  shop  by  the  engine.  The  incumbent  may  NOT  use 
the  PUB.  Hand  the  incumbent  the  P&D  valve  before  you  read  the 
instructions. 


SAY  TO  THE  INCUMBENT 

WITHOUT  THE  USE  OF  THE  PUB,  TELL  ME  STEP  BY  STEP  HOW  YOU  SHOULD 
INSTALL  THE  P&D  VALVE.  INCLUDE  ALL  SAFETY  PRECAUTIONS  THAT  YOU 
SHOULD  FOLLOW.  YOU  MAY  USE  THE  ENGINE,  THE  P&D  VALVE,  AND  ANY 
TOOLS  THAT  YOU  WOULD  NORMALLY  USE  TO  SHOW  ME  HOW  THE  VALVE  SHOULD 
BE  INSTALLED. 


Performed  or  Answered  Correctly  Yes  No 

Did  the  incumbent  say  he/she  would; 

1.  Lubricate  the  0-rings  and  seal  with 

petrolatum  prior  to  installation?  _  _ 

2.  Install  the  large  elbows,  jam  nuts  and 
0-rings  in  the  large  outlet  ports  and 

rear  port?  _  _ 

3.  Leave  the  jam  nuts  finger  tight  until 

after  the  fuel  manifolds  were  Installed?  _  _ 
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RATER  TRAINING  PROGRAM 


Adainlstrator* s  Guide 

HAND  OUT  RATER  TRAINING  BOOKLETS,  THEN  SAY:  WE  ARE  GOING  TO  BEGIN  THE  RATF.R 
TRAINING  SESSION.  PLEASE  TURN  TO  PAGE  1  AND  FOLLOW  ALONG  WHILE  I  READ  OUT 
LQUP. _ PLEASE  DO  NOT  MARK  IN  THE  BOOKLETS. 

I .  Introduction 

For  the  next  several  hours,  your  time  will  be  spent  focusing  on  the  rating 
forms  you  will  complete  as  part  of  this  research  project.  You  will  use  several 
rating  forms  to  rate  the  performance  of  yourself,  your  coworkers,  or  if  you  are 
a  supervisor,  your  subordinates. 

Before  you  use  any  of  the  rating  forms,  we  are  going  to  talk  about  each  form, 
its  purpose,  and  how  to  use  each  form  to  effectively  rate  an  individual.  We 
are  also  going  to  discuss  some  ideas  that  will  help  you  use  the  rating  forms 
and  make  the  most  accurate  ratings  possible. 

It  is  essential  to  the  outcome  of  this  project  that  you  be  truthful  and  honest 
in  your  ratings.  The  ratings  will  not  be  seen  by  your  coworkers,  supervisor 
or  anyone  else  connected  with  your  unit.  The  data  collected  will  be  seen  only 
by  Navy  Personnel  Research  and  Development  personnel  and  the  private  contractor 
associated  with  this  project.  The  information  you  provide  will  be  coded  to 
assure  confidentiality  and  the  rating  forms  will  subsequently  be  destroyed. 
The  ratings  will  be  used  for  research  purposes  only  and  will  in  no  wav  effect 
anyone's  career.  Therefore,  please  rate  each  person  as  accurately  as  possible. 
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Performance  Factor  1  (PFl):  Technical  Knovledge/Sklll 


Displaying  Job  knowledge  and  skill. 


Levels 


Circle  the 

Number  Behavioral  Examples 


Always  exceeds 
acceptable  level 
of  performance 


5  Displays  exceptional  knowledge/skill 

to  consistently  complete  assignments 
and  tasks  properly;  requires  little  or 
no  supervision;  completes  tasks  in 
minimum  time. 


Frequently  exceeds 
acceptable  level 
of  performance 


4  Displays  considerable  knowledge  and 

skill  to  complete  assignments  and  casks 
properly;  performs  effectively  with 
little  supervision;  completes  tasks 
quicker  than  average  first-term  sailors 
or  marines. 


Meets  acceptable 
level  of 
performance 


Occasionally  meets 
acceptable  level 
of  performance 


Never  meets 
acceptable  level 
of  performance 


3  Displays  good  knowledge/skill  in  most 

aspects  of  the  job;  able  to  properly 
complete  the  majority  of  tasks; 
requires  supervision  only  on  difficult 
tasks  and  assignments;  completes  work 
in  the  same  time  as  other  first -term 
sailors  or  marines. 

2  Occasionally  displays  adequate  know¬ 

ledge  about  how  to  complete  tasks  and 
assignments;  quality  of  work  Is  incon¬ 
sistent;  requires  direct  supervision  on 
most  tasks  to  ensure  quality  and 
accuracy;  usually  completes  tasks 
within  required  time. 

1  Does  not  display  knowledge  and  skill 

necessary  to  properly  complete  tasks 
and  assignments;  unable  to  perform 
without  direct  supervision;  often  falls 
to  complete  assignments;  performs 
slower  than  other  first- term  sailors  or 
marines. 
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SCENARIO  FOR  TASK  360:  You  have  been  Instructed  to  Install  the  exhaust  gas 
thermocouple  harness. 

9.  (1052)  Which  of  the  components  (A,B,C)  in  the  picture  to  the  left  Is  the 
EGT  thermocouple  harness .  Write  the  matching  letter  on  the  answer  sheet . 

10.  From  the  list  below,  select  the  actions  or  checks  that  you  should  cake 
when  installing  the  EGT  thermocouple  harness.  Place  a  checkmark  in  the  yes 
column  of  the  answer  sheet  if  you  should  perform  the  action.  Place  a  checkmark 
in  the  no  column  if  you  should  not  perform  the  action. 

(1053)  Ensure  that  no  burrs  remain  on  the  threads  or  mating  surface  of  the 
turbine  frame  boss. 

(1054)  Ensure  that  no  lubricant  ran  Into  the  filler  material  around  the 
thermocouple  loops  at  the  ends  of  the  probes . 

(1055)  Apply  a  thin  coating  of  lubricant  on  the  threaded  areas  of  each 
thermocouple  boss  on  the  turbine  frame. 

(1056)  Torque  each  nut  as  each  probe  Is  Installed. 

(1057)  Ensure  that  the  harness  did  not  bend  causing  the  Inside  filament  to 
be  broken. 

(1058)  Install  a  copper  gasket  between  the  top  harness  and  the  lead. 

(1059)  Connect  the  forward  end  of  each  harness  to  the  rigid  thermocouple 
lead. 

(1060)  Safety  wire  eech  thermocouple  harness  mounting  boss  to  a  turbine  case 
flange  bolt. 
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GENERAL  UTILITY/ACCEPTABILITY  QUESTIONNAIRE  FOR 
THE  NAVY/MARINE  CORPS  J-79  JET  ENGINE  MECHANIC 
PERFORMANCE  ASSESSMENT  SYSTEM 


NAME _ SSN _ 

We  are  interested  in  your  beliefs  about  the  usefulness  of  the 
entire  performance  measurement  system  (i.e.,  rating  forms,  hands 
on  testing,  interview  testing,  and  job  knowledge  test  with  photo¬ 
graphs)  as  well  as  your  beliefs  about  how  the  performance 
information  will  be  used.  Please  respond  to  questions  1  through 
6  by  using  the  scale  below. 

1  =  Not  at  all 

2  =  To  a  small  extent 

3  =  To  a  moderate  extent 

4  =  To  a  great  extent 

5  =  To  a  very  great  extent 


1.  How  much  do  you  believe  each  measuring  instrument  will 
allow  someone  to  correctly  determine  your  level  of  job 
proficiency? 

_  Rating  Forms 

_  Hands-On  Tests 

_  Interview  Tests 

_  Job  Knowledge  Test 

2.  How  acceptable  (i.e.,  easy  to  use  and  understand)  do  you 
believe  each  measuring  instrument  is  as  a  means  of  determining 
job  proficiency? 

_  Rating  Forms 

_  Hands-On  Tests 

_  Interview  Tests 

_  Job  Knowledge  Tests 

3.  How  motivated  were  you  to  complete  each  measuring  session? 
_  Rating  Forms 

_  Hands-On  Tests 

_  Interview  Tests 

_  Job  Knowledge  Tests 
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